Image processing apparatus image processing method and storage medium for lighting processing on image using model data

ABSTRACT

An image processing apparatus generates a normal image, when lighting correction of an image is performed using three-dimensional shape data. Three-dimensional shape data of a predetermined object is adjusted with a subject in position, based on a feature pattern detected in image data, and lighting processing for correcting a pixel value of the image data is performed based on the adjusted three-dimensional shape data, distance information of the subject, and a position of a virtual light source. When the three-dimensional shape data is adjusted with the image data in position, a region having a large irregularity in the three-dimensional shape data is preferentially adjusted in position.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Divisional of U.S. application Ser. No.15/093,585, filed Apr. 7, 2016, which claims priority from JapanesePatent Application No. 2015-083724, filed Apr. 15, 2015, and No.2015-102819, filed May 20, 2015, which are hereby incorporated byreference herein in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

One disclosed aspect of the embodiments relates to a technique foradding shade to image data.

Description of the Related Art

When an image of a subject is captured using an imaging apparatus ordevice, illumination conditions, such as a light amount and a directionof light for the subject, greatly influence the appearance of thecaptured image. For example, when light is obliquely incident upon asubject, a three-dimensional effect of the subject is emphasized. Whenlight is incident upon a subject from the rear, the front of the subjectis in shadow and therefore, an image gives a dim and flat impression. Asa method for adjusting the appearance of such an image, Japanese PatentNo. 5088220 discusses a method for generating an image under virtualillumination conditions by using three-dimensional shape datacorresponding to the subject. More specifically, in the techniquediscussed in Japanese Patent No. 5088220, an image under virtualillumination conditions is rendered using three-dimensional shape data,and then this image is placed in a region where a subject is present ina captured image, so that an image under different illuminationconditions is generated. As another method, Japanese Patent No. 5147287discusses a method for estimating a normal direction of a surface of asubject from a captured image, and generating an image under virtualillumination conditions based on the estimated normal direction. Morespecifically, in the technique discussed in Japanese Patent No. 5147287,the normal direction is estimated from brightness information of thecaptured image, and an image, in which lighting is corrected based onthe normal direction, is generated.

However, the technique discussed in Japanese Patent No. 5088220 has thefollowing problem. In this technique, the image rendered from thethree-dimensional shape data is a computer graphics (CG) image, and thisCG image and the captured image are synthesized. Therefore, if positionadjustment between the CG image and the captured image is poor, astrange image is obtained. The technique discussed in Japanese PatentNo. 5147287 also has a problem. In this technique, the normal directionis estimated based on the brightness of the captured image andtherefore, the correct normal direction cannot be estimated if noise andshadow are included in the subject image.

SUMMARY OF THE INVENTION

One disclosed aspect of the embodiments is directed to an imageprocessing apparatus capable of generating a normal image, when lightingcorrection of an image is performed using three-dimensional shape data.

According to an aspect of the embodiments, an image processing apparatusincludes an image acquisition unit, a distance acquisition unit, asetting unit, a detection unit, a holding unit, a position adjustmentunit, a position adjustment unit, and a processing unit. The imageacquisition unit is configured to acquire image data including asubject. The distance acquisition unit is configured to acquire distanceinformation, which indicates a distance from an imaging device that hascaptured the image data to the subject, for each pixel of the imagedata. The setting unit is configured to set a position of a lightsource, in a scene indicated by the image data. The detection unit isconfigured to detect, in the image data, a predetermined feature patternof the subject, and to detect a position of each of a plurality ofreference points to be used for position adjustment of data, based onthe feature pattern. The holding unit is configured to hold model dataindicating a three-dimensional shape of a predetermined object havingthe predetermined feature pattern, and to hold information indicating aposition of each of the plurality of reference points in the model data.The position adjustment unit is configured to perform positionadjustment of the model data with the subject, based on the position ofeach of the plurality of reference points included in the model data,and the position of each of the plurality of reference points detectedby the detection unit. The processing unit is configured to performlighting processing for correcting a pixel value of the image data,based on the model data for which the position adjustment has beenperformed by the position adjustment unit, the distance information, andthe position of the light source, wherein, in the position adjustment,the position adjustment unit places greater importance on a degree ofagreement in terms of a first reference point among the plurality ofreference points, than on a degree of agreement in terms of a secondreference point different from the first reference point.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams illustrating a housing of an imagingapparatus or device according to a first exemplary embodiment.

FIG. 2 is a block diagram illustrating an internal configuration of theimaging apparatus or device according to the first exemplary embodiment.

FIG. 3 is a block diagram illustrating a configuration of an imageprocessing unit according to the first exemplary embodiment.

FIG. 4 is a flowchart illustrating an operation of the image processingunit according to the first exemplary embodiment.

FIGS. 5A, 5B, and 5C are diagrams each illustrating an example of imagedata according to the first exemplary embodiment.

FIG. 6 is a diagram illustrating an example of a positional relationshipbetween the imaging apparatus or device, and a subject as well asillumination, according to the first exemplary embodiment.

FIG. 7 is a diagram illustrating an example of image data after lightingcorrection processing according to the first exemplary embodiment.

FIG. 8 is a block diagram illustrating a configuration of a normalacquisition unit according to the first exemplary embodiment.

FIG. 9 is a flowchart illustrating an operation of the normalacquisition unit according to the first exemplary embodiment.

FIG. 10 is a diagram illustrating examples of face model data accordingto the first exemplary embodiment.

FIGS. 11A and 11B are diagrams illustrating an example of each organposition in captured image data, and an example of an irregular regionin face model data, respectively, according to the first exemplaryembodiment.

FIGS. 12A, 12B, and 12C are diagrams illustrating an example ofcorrection processing for normal image data by using a cross bilateralfilter according to the first exemplary embodiment.

FIG. 13 is a block diagram illustrating a configuration of an imageprocessing unit according to a second exemplary embodiment.

FIG. 14 is a flowchart illustrating an operation of the image processingunit according to the second exemplary embodiment.

FIGS. 15A and 15B are diagrams illustrating an example of captured imagedata, and an example of face model data after projective transformation,respectively, according to the second exemplary embodiment.

FIGS. 16A and 16B are diagrams illustrating an example of normal imagedata, and an example of mask image data, respectively, according to thesecond exemplary embodiment.

FIG. 17 is a block diagram illustrating a configuration of an imageprocessing unit according to a third exemplary embodiment.

FIG. 18 is a flowchart illustrating an operation of the image processingunit according to the third exemplary embodiment.

FIGS. 19A, 19B, 19C, and 19D are diagrams illustrating an example ofcaptured image data, an example of range image data, an example ofnormal image data, and an example of normal image data, respectively,according to the third exemplary embodiment.

FIGS. 20A and 20B are diagrams each illustrating an external appearanceof an imaging apparatus or device according to a fourth exemplaryembodiment.

FIG. 21 is a block diagram illustrating an internal configuration of theimaging apparatus device according to the fourth exemplary embodiment.

FIG. 22 is a block diagram illustrating a configuration of an imageprocessing unit according to the fourth exemplary embodiment.

FIG. 23 is a flowchart illustrating a flow of processing according tothe fourth exemplary embodiment.

FIGS. 24A and 24B are diagrams each illustrating an example of imagedata according to the fourth exemplary embodiment.

FIG. 25 is a diagram illustrating face information according to thefourth exemplary embodiment.

FIG. 26 is a flowchart illustrating a flow of distance correctionprocessing according to the fourth exemplary embodiment.

FIGS. 27A, 27B, 27C, and 27D are diagrams illustrating an outline of thedistance correction processing according to the fourth exemplaryembodiment.

FIG. 28 is a flowchart illustrating a flow of normal correctionprocessing according to the fourth exemplary embodiment.

FIGS. 29A and 29B are diagrams illustrating an outline of the normalcorrection processing according to the fourth exemplary embodiment.

FIGS. 30A and 30B are diagrams each illustrating an outline of thenormal smoothing processing according to the fourth exemplaryembodiment.

FIG. 31 is a flowchart illustrating a flow of lighting processingaccording to the fourth exemplary embodiment.

FIG. 32 is a diagram illustrating an outline of the lighting processingaccording to the fourth exemplary embodiment.

FIG. 33 is a diagram illustrating an effect of the lighting processingaccording to the fourth exemplary embodiment.

FIGS. 34A and 34B are graphs each illustrating an example of acorrection coefficient according to a fifth exemplary embodiment.

FIG. 35 is a diagram illustrating an effect of lighting processingaccording to a sixth exemplary embodiment.

FIG. 36 is a flowchart illustrating a flow of processing according to aseventh exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the disclosure will be described below withreference to the drawings. The following exemplary embodiments are notintended to limit the disclosure. In addition, not all combinations offeatures described in the exemplary embodiments are necessary for thedisclosure. Similar configurations are provided with the same referencenumerals, when described.

FIGS. 1A and 1B are diagrams illustrating a housing of an imagingapparatus or device 101 according to a first exemplary embodiment. FIG.1A illustrates a front face of the imaging apparatus 101, and FIG. 1Billustrates a back face of the imaging apparatus 101. The housing of theimaging apparatus 101 includes two imaging units 102 and 103, an imagingbutton 104, a light emitting unit 105, a display unit 106, and anoperation unit 107. The imaging units 102 and 103 are each configured ofan imaging optical system including components such as a zoom lens, afocus lens, a shake correction lens, an iris, a shutter, an opticalfilter, a sensor, an analog-to-digital (A/D) conversion unit. Theimaging units 102 and 103 each receive light information of a subjectwith the sensor. The sensor is configured of an imaging sensor such as acomplementary metal oxide semiconductor (CMOS) sensor and a chargecoupled device (CCD) sensor. The light information received by thesensor is converted into digital data by the A/D conversion unit. Theimaging button 104 is provided for a user to instruct the imagingapparatus 101 to start image capturing. The light emitting unit 105 isan illuminator capable of emitting light when image capturing startsaccording to an instruction of the user. The display unit 106 is adisplay for displaying, for example, image data processed by the imagingapparatus 101 and a setting menu. For example, a liquid crystal displaycan be used for the display unit 106. The operation unit 107 is a buttonprovided for the user to specify a changeable parameter in the imagingapparatus 101. The display unit 106 may have a touch screen function,and in such a case, a user instruction provided using the touch screencan be treated as an input to the operation unit 107.

FIG. 2 is a block diagram illustrating an internal configuration of theimaging apparatus 101. A central processing unit (CPU) 201 is aprocessing circuit that comprehensively controls each part to bedescribed below. A random access memory (RAM) 202 serves as a mainmemory and a work area for the CPU 201. A read only memory (ROM) 203stores a control program to be executed by the CPU 201. A bus 204 is atransfer path for various kinds of data. For example, digital dataacquired by each of the imaging units 102 and 103 is sent to apredetermined processing unit via the bus 204. A display control unit205 is a control circuit that performs display control for image dataand characters to be displayed on the display unit 106. An imagingcontrol unit 206 is a drive circuit that performs control, such asfocusing, opening/closing of the shutter, and adjustment of an aperture,for each of the imaging units 102 and 103, based on an instruction fromthe CPU 201. A digital signal processing unit 207 is a processingcircuit that performs various kinds of processing, such as white balanceprocessing, gamma processing, and noise reduction processing, on digitaldata received via the bus 204. An encoding unit 208 is a processingcircuit that converts digital data into a file format such as JointPhotographic Experts Group (JPEG) and Motion Picture Experts Group(MPEG). An external memory control unit 209 is an interface forconnecting the imaging apparatus 101 to a personal computer (PC) and amedium (e.g., a hard disk, a memory card, a CompactFlash (CF) card, aSecure Digital (SD) card, a Universal Serial Bus (USB) memory). An imageprocessing unit 210 is a processing circuit that performs imageprocessing such as lighting correction, by using digital data acquiredby the imaging units 102 and 103, or digital data output from thedigital signal processing unit 207. The image processing unit 210 willbe described in detail below. Components other than those describedabove included in the imaging apparatus 101 are not the main point ofthe present exemplary embodiment and therefore will not be described.

<Image Processing Unit>

FIG. 3 is a block diagram illustrating a functional configuration of theimage processing unit 210. The image processing unit 210 serves as eachblock illustrated in FIG. 3, when being controlled by the CPU 201executing a program stored in the ROM 203. FIG. 4 is a flowchartillustrating the operation of the image processing unit 210. Processingto be performed by the image processing unit 210 will be described belowwith reference to FIGS. 3 and 4.

In step S401, an image acquisition unit 301 acquires captured image datarepresenting an image captured by the imaging unit 102. The imageacquisition unit 301 outputs the acquired captured image data to alighting unit 305.

Next, in step S402, a distance acquisition unit 302 acquires range imagedata. The range image data is data indicating a distance from a subjectto the imaging apparatus 101, at each pixel position in image data. Inthe present exemplary embodiment, the range image data is a 256-stepgradation monochrome image. In the range image data, the pixel value ofa pixel corresponding to a subject at a short distance from an imagingapparatus is large, and the pixel value of a pixel corresponding to asubject at a long distance from an imaging apparatus is small. Theformat of the range image data is not limited to this, and may be, forexample, a table storing distance information of a subject at each pixelposition.

Now, a method for acquiring the range image data will be described. Therange image data can be acquired using a stereo matching method. Thestereo matching method detects corresponding points of a subject thatare identical object points, from pieces of image data of differentviewpoints, and calculates a distance by the principle of triangulation,based on the difference between the corresponding points in therespective pieces of image data. Other usable methods include a time offlight (TOF) method and a pattern projection method. The TOF methodcalculates a distance, based on the time from emission of light toward asubject to receipt of the light reflected from the subject. The patternprojection method emits light of a pattern to a subject, and calculatesa distance based on a displacement of the pattern. In the presentexemplary embodiment, it is assumed that the range image data isacquired by the TOF method. The imaging apparatus 101 causes the lightemitting unit 105 to emit light toward a subject at a start of imagecapturing, and the imaging unit 103 receives the light reflected fromthe subject. The range image data is generated based on the time fromthe emission of the light to the receipt of the light reflected from thesubject. The range image data may be acquired by a known method otherthan the above-described methods. The distance acquisition unit 302generates the range image data based on the data input from the imagingunit 103, and outputs the generated range image data to the lightingunit 305.

Next, in step S403, a normal acquisition unit 303 calculates normalimage data. The normal image data is data indicating a normal vectorwith respect to a surface of a subject, at each pixel position in imagedata. In the present exemplary embodiment, the normal image data is a256-step gradation color image, and expresses a direction vector definedin a coordinate system including a subject, as a pixel value. The formatof the normal image data is not limited to this, and may be, forexample, a table storing normal information of a subject in each pixel.

Now, a method for calculating the normal image data will be described.The normal image data can be calculated by a method of performingdifferentiation with respect to a target pixel of the range image datafrom a neighboring region, or a method of applying a plane. The normalimage data can also be calculated by a method of transforming normalimage data of a typical face shape stored beforehand in the ROM 203,according to a shape or facial expression of a subject. In the presentexemplary embodiment, it is assumed that normal image data of a typicalface shape is stored in the ROM 203, and the normal image data iscalculated by the method of transforming the normal image data of thetypical face shape, according to a subject. This normal image datageneration method will be described in detail below.

FIGS. 5A, 5B, and 5C are diagrams each illustrating an example of imagedata in the present exemplary embodiment. FIG. 5A illustrates thecaptured image data acquired in step S401. FIG. 5B illustrates the rangeimage data acquired in step 402. FIG. 5C illustrates the normal imagedata calculated in step S403. Here, the captured image data is definedas I, the range image data is defined as P, the normal image data isdefined as N, and the respective values corresponding to a pixel (i, j)are assumed to be I(i, j), P(i, j), and N(i, j). Further, the capturedimage data and the normal image data are both color image data, and thusexpressed as I(i, j)=(Ir(i, j), Ig(i, j), Ib(i, j)), and N(i, j)=(Nx(i,j), Ny(i, j), Nz(i, j)), respectively. The normal acquisition unit 303outputs the acquired normal image data to the lighting unit 305.

Next, in step S404, an illumination acquisition unit 304 acquiresillumination parameters of virtual illumination to be use for lightingprocessing. The illumination parameter is a variable representing eachof a color of illumination, brightness of the illumination, and aposition as well as an orientation of the illumination. In the presentexemplary embodiment, it is assumed that the color of the illuminationis 256-step gradation RGB values, and the brightness of the illuminationis a positive real number value. Further, each of the position and theorientation of the illumination is a three-dimensional real numbervalue, and expressed as coordinates and a direction vector in a spacedefined in a coordinate system including a subject. Here, the color ofthe illumination is defined as L, the brightness of the illumination isdefined as α, the position of the illumination is defined as Q, and theorientation of the illumination is defined as U. It is assumed that thecolor of the illumination, the position of the illumination, and theorientation of the illumination are expressed as L=(Lr, Lg, Lb), Q=(Qx,Qy, Qz), and U=(Ux, Uy, Uz), respectively.

Now, a positional relationship between an imaging apparatus, and asubject as well as illumination will be described with reference to FIG.6. FIG. 6 illustrates a virtual space including an imaging apparatus601, a subject 602, and an illuminator 603. A rightward direction is anX-axis, a frontward direction is a Y-axis, and an upward direction is aZ-axis. The imaging apparatus 601 represents a virtual sensor plane, andthe value I(i, j) corresponding to the pixel (i, j) is defined in amanner illustrated in FIG. 6. Similarly, the normal N(i, j) and thedistance P(i, j) each corresponding to the pixel (i, j) are defined in amanner illustrated in FIG. 6. The illuminator 603 is a virtual lightsource that is not real, and the position Q and the orientation U of theillumination are defined in a manner illustrated in FIG. 6. Theillumination acquisition unit 304 sets the acquired illuminationparameters, as illumination parameters to be used for the lightingprocessing, and outputs the illumination parameters to the lighting unit305.

Lastly, in step S405, the lighting unit 305 performs lighting correctionprocessing on the image data input from the image acquisition unit 301.The lighting correction processing is processing for correcting a pixelvalue of the captured image data by using the range image data and thenormal image data, as if lighting based on the illumination parametersis applied to the subject. In the present exemplary embodiment, it isassumed that the captured image data is corrected according to anexpression (1).

Ir′(i, j)=Ir(i, j)+kr(i, j)×Lr×Ir(i, j)

Ig′(i, j)=Ig(i, j)+kg(i, j)×Lg×Ig(i, j)

Ib′(i, j)=Ib(i, j)+kb(i, j)×Lb×Ib(i, j)   (1)

Here, image data after the image processing is defined as I′, and avalue corresponding to the pixel (i, j) is therefore I′(i, j), andexpressed as I′(i, j)=(Ir′(i, j), Ig′(i, j), Ib′(i, j)). Further, k(i,j) can be calculated, for example, by an expression (2), based on thebrightness, the position, and the orientation of the illumination, aswell as the distance information and the normal informationcorresponding to the pixel (i, j).

k(i, j)=t×α×K(ρ)×(N(i, j),V(i, j))/(W(P(i, j)),Q))   (2)

Here, t is an arbitrary constant, and ρ is an angle formed by theorientation U of the illumination, and a straight line, which connectsthe position of the subject corresponding to the pixel (i, j) with theposition Q of the illumination. K(ρ) is a function returning a valuethat increases with decrease in the angle ρ. Further, V(i, j) is a unitvector that represents a direction from the position of the subjectcorresponding to the pixel (i, j) to the position Q of the illumination.W(P(i, j)),Q) is a function returning a value that increases withincrease in the distance from the position of the subject correspondingto the pixel (i, j) to the position Q of the illumination. Theexpression to be used for the lighting processing is not limited to anexpression of performing addition for a pixel value as a distance from alight source becomes shorter, such as the one described above.Alternatively, there may be used, for example, an expression ofperforming subtraction for a pixel value of captured image data as adistance from a light source becomes longer. If such an expression isused, natural shade can be added to image data that results from imagecapturing performed in a state with an excessive light amount.

FIG. 7 is a diagram illustrating an example of image data after thelighting correction processing according to the present exemplaryembodiment. Shade that produces a three-dimensional effect, as if anobject is lighted with illumination from a desired position, can beadded to the captured image data illustrated in FIG. 5A.

An outline of the processing performed in the imaging apparatus 101according to the present exemplary embodiment has been described above.The above-described processing can generate, from the captured imagedata, image data representing an image as if the image is captured underillumination conditions desired by a user. Therefore, even if imagecapturing is performed under unsuitable illumination conditions such asspotlight illumination and backlight, it is possible to generate imagedata in which the illumination conditions are changed to suitableillumination conditions after the image capturing. The processing (stepS403) performed by the normal acquisition unit 303 will be described indetail below.

<Normal Acquisition Unit>

Next, a method for calculating the normal information by transformingthe normal image data of the typical face shape according to the subjectwill be described. FIG. 8 is a block diagram illustrating a functionalconfiguration of the normal acquisition unit 303. Further, FIG. 9 is aflowchart illustrating an operation of the normal acquisition unit 303.Details of the processing (step S403) performed by the normalacquisition unit 303 will be described, with reference to the blockdiagram illustrated in FIG. 8 and the flowchart illustrated in FIG. 9.

First, in step S901, an image acquisition unit 801 acquires the capturedimage data acquired in step S401. The image acquisition unit 801 outputsthe acquired captured image data to a pattern detection unit 802 and anormal correction unit 806.

Next, in step S902, the pattern detection unit 802 detects a featurepattern of the subject. In the present exemplary embodiment, the patterndetection unit 802 detects a human face from the captured image data. Inthe present exemplary embodiment, it is assumed that the ROM 203 storesbeforehand a database generated by learning a feature of a face image,and the pattern detection unit 802 detects a face by, for example,template matching, using the database stored in the ROM 203. The featurepattern detected here is not limited to the face, and various objectssuch as a character written on a subject can be used. The patterndetection unit 802 acquires subject information indicating an organposition of each part such as eyes and a nose and a direction of theface in the captured image data, and outputs the subject information toa normal selection unit 803 and a normal generation unit 805. The methodfor acquiring the subject information is not limited to this method. Forexample, the user may input organ positions and a direction of the facewhile viewing the captured image.

Next, in step S903, the normal selection unit 803 selects normal imagedata to be used for the lighting processing, from the pieces of normalimage data of the typical face shape stored in the ROM 203. The normalimage data of the typical face shape will be hereinbelow referred to as“face model data”. In the present exemplary embodiment, the face modeldata, which corresponds to each of a plurality of different directionsof the face, is stored beforehand in the ROM 203. Based on the directionof the face detected in step S902, the normal selection unit 803 selectsthe face model data indicating the closest direction of the face to thedirection of the detected face. FIG. 10 is a diagram illustratingexamples of the face model data. In FIG. 10, the face model data (eachof data 1001 to data 1005) is generated so that an angle is changed insuch a manner that the highest positions of a nose are laterallyarranged at equal intervals in the data 1001 to data 1005. When thetypical face shape is an oval shape having a protruding nose, a noseregion around the nose changes most according to the change of theangle, in the face model data. Therefore, it is desirable to provide aplurality of pieces of data as the face model data based on the positionof a nose. However, if such face model data cannot be prepared, theremay be provided face model data including a plurality of pieces of datacorresponding to the respective faces arranged at equal angles. Thenormal selection unit 803 outputs the selected face model data to anirregularity determination unit 804.

Next, in step S904, the irregularity determination unit 804 determinesan irregular region and an outline region of the face model data. Theirregular region of the face model data is a region with a steep changein the face model data, and corresponds to an organ region of a partsuch as an eye and a nose, when a face region is a target. In thepresent exemplary embodiment, it is assumed that the ROM 203 storesbeforehand an irregular region and an outline region corresponding toeach of the face model data, and the stored data is read out. Forexample, the ROM 203 stores positions of a nose, eyes, and a mouth ineach of the face model data. A part around the nose in particular has alarge irregularity. Therefore, it is assumed that this part is theirregular region. It is assumed that pixel positions corresponding tothe outermost part of a face in the face model data are stored as theoutline region. The method for determining the irregular region and theoutline region is not limited to this method. For example, the irregularregion may be determined using a database as in step 902. Alternatively,the irregular region may be determined as follows. First, the face modeldata is differentiated with respect to each of an x component, a ycomponent, and a z component. Subsequently, a region, in which thedegree of a change in at least one of these three components exceeds apredetermined threshold, is determined as the irregular region. Stillalternatively, the irregular region may be determined as follows. First,the face model data is differentiated in a range of a predeterminednumber of pixels around a point determined as a reference point forposition adjustment, and then, the size of a shape change around thereference point is derived by differentiating the face model data. Basedon the size of the derived shape change, an organ position having alarge irregularity is determined as the irregular region. Theirregularity determination unit 804 outputs information indicating theface model data, the organ positions in the face model data, and theirregular region, to the normal generation unit 805.

In step S905, the normal generation unit 805 generates the normal imagedata, based on the irregular region determined in step S904. In thepresent exemplary embodiment, projective transformation of the facemodel data is performed, based on the organ positions in the faceindicated by the subject information input from the pattern detectionunit 802, and the organ positions as well as the irregular region in theface model data input from the irregularity determination unit 804.FIGS. 11A and 11B are diagrams illustrating examples of the organposition in the captured image data and that in the face model data,respectively. In the captured image data illustrated in FIG. 11A, aright eye 1101, a left eye 1102, a nose 1103, and a mouth 1104 are thedetected organ positions. In the face model data illustrated in FIG.11B, a right eye 1105, a left eye 1106, a nose 1107, and a mouth 1108are the detected organ positions, and the nose 1107 is determined as theirregular region. Based on a correspondence relationship between theseorgan positions, a transformation matrix for performing the projectivetransformation of the face model data can be calculated.

The normal generation unit 805 generates the transformation matrix forthe projective transformation by using a least squares method based on adifference between the organ positions, in such a manner that the organposition in the captured image data and the corresponding organ positionin the face model data become close to each other. In other words, theorgan positions are each determined as a reference point to be used forposition adjustment, and the normal generation unit 805 performs theposition adjustment between the captured image data and the face modeldata, to reduce the difference between the reference point in thecaptured image data and the corresponding reference point in the facemodel data. In this process, the normal generation unit 805 calculatesthe transformation matrix by using the least squares method in which ahigh weight is assigned to the position of the nose so that the positionof the nose determined as the irregular region and the position of thenose in the captured image data become closest to each other. In otherwords, a larger value is assigned to the difference between thepositions of the noses, as an evaluation value indicating thedifference. This means that the normal generation unit 805 placesgreater importance on the degree of agreement between the positions ofthe noses than on the degree of agreement between the othercorresponding organ positions, in the generation of the transformationmatrix. This is because the nose region has the largest change in thenormal in the face region, and if there is a difference between thepositions of the noses, a high degree of strangeness or abnormality mayappear after the lighting processing. The above-described processing canreduce strangeness that appears in an image after the lightingprocessing, even if there is a positional difference betweenthree-dimensional shape data and captured image data.

In the present exemplary embodiment, the face model data is subjected tothe projective transformation to match with the captured image data.However, the position adjustment between the face model data and thecaptured image data is not limited to the projective transformation, andtranslation may be adopted. In the case where the translation isadopted, the normal generation unit 805 determines an amount ofdisplacement for normal model data, by using a weighted least squaresmethod based on a distance of each organ position. According to thisprocessing, a workload can be reduced to be less than a workload in thecase where the transformation matrix of the projective transformation isdetermined. For example, this processing may be sufficient for a casewhere the face model data is stored using a small angle variation forevery angle. The normal generation unit 805 generates the normal imagedata in which the face model data is arranged at the correspondingposition in the captured image data, and outputs the generated normalimage data to the normal correction unit 806. In the normal image data,it is assumed that N(i, j)=(0, 1, 0) is stored in all the regions notcorresponding to the face model data.

Lastly, in step S906, the normal correction unit 806 corrects the normalimage data, based on the captured image data, and the outline region ofthe face model data. In the present exemplary embodiment, the normalimage data is corrected by causing a cross bilateral filter based on thepixel value of the captured image data to act on the normal image data.The cross bilateral filter is a smoothing filter for assigning highweights to the pixels having the pixel values close to each other in thecaptured image data. The weight to each pixel in filtering processing isdetermined based on the pixel value of the captured image data andtherefore, the normal image data is corrected to a shape closer to thatof the captured image data. FIGS. 12A, 12B, and 12C are diagramsillustrating how a processing result is obtained by the cross bilateralfilter. More specifically, FIG. 12A illustrates the captured image data,FIG. 12B illustrates the normal image data before the processing, andFIG. 12C illustrates the normal image data after the processing. Asillustrated in FIG. 12C, it can be seen that the normal image data canbe transformed to match with the outline of the captured image data, byfiltering the normal image data illustrated in FIG. 12B with referenceto the captured image data illustrated in FIG. 12A. The normalcorrection unit 806 outputs the normal image data for which thecorrection is completed, to the lighting unit 305, and then theprocessing ends. The filter used here is not necessarily the crossbilateral filter, and any type of filter may be used if the filter is asmoothing filter based on the pixel value of the captured image data.The outline of the processing performed in the normal acquisition unit303 has been described above. According to the above-describedprocessing, strangeness or slight abnormality that appears in an imageafter the lighting processing can be reduced, even if there is apositional difference between three-dimensional shape data and capturedimage data.

In the present exemplary embodiment, the image acquisition unit 301serves as an image acquisition unit configured to acquire image dataincluding a subject. The distance acquisition unit 302 serves as adistance acquisition unit configured to acquire distance information,which indicates a distance from an imaging apparatus capturing an imagerepresented by the image data to the subject, for each pixel of theimage data. The illumination acquisition unit 304 serves as a settingunit configured to set a position of a light source, in a sceneindicated by the image data. The pattern detection unit 802 serves as adetection unit configured to detect, in the image data, a predeterminedfeature pattern of the subject, and to detect a position of each of aplurality of reference points to be used for position adjustment ofdata, based on the feature pattern.

Further, the ROM 203 serves as a holding unit configured to hold modeldata indicating a three-dimensional shape of a predetermined objecthaving the predetermined feature pattern, and to hold informationindicating a position of each of the plurality of reference points inthe model data. The normal generation unit 805 serves as a positionadjustment unit configured to perform position adjustment of the modeldata with the subject, based on the position of each of the plurality ofreference points included in the model data, and the position of each ofthe plurality of reference points detected by the detection unit. Thelighting unit 305 serves as a processing unit configured to performlighting processing for correcting a pixel value of the image data,based on the model data for which the position adjustment is performedby the position adjustment unit, the distance information, and theposition of the light source.

Further, the normal selection unit 803 serves as a selection unitconfigured to select model data corresponding to one of the plurality ofdirections, based on the position of each of the plurality of referencepoints in the model data, and the position of each of the plurality ofreference points detected by the detection unit.

Further, the irregularity determination unit 804 serves as adetermination unit configured to determine the first reference point andthe second reference point from the plurality of reference points.

In the first exemplary embodiment, there is described the method forsuppressing strangeness of the image data after the lighting processing,by preferentially registering the region having the large irregularityin the face model data. In a second exemplary embodiment, there will bedescribed an example in which the normal image data is corrected tomaintain a three-dimensional effect of an outline region of a subject byalso using mask image data.

FIG. 13 is a block diagram illustrating a configuration of the normalacquisition unit 303 according to the second exemplary embodiment.Further, FIG. 14 is a flowchart illustrating an operation of the normalacquisition unit 303 according to the second exemplary embodiment.Processing from step S901 to step S904 is similar to the processing fromstep S901 to step S904 in the first exemplary embodiment, and thereforewill not be described. Brief description will be provided, focusing on apoint different from the first exemplary embodiment.

In step S1401, a normal generation unit 1301 generates normal image databased on the irregular region determined in step S904. In the firstexemplary embodiment, there is described the method for performing theprojective transformation of the face model data, based on the organpositions of the face indicated by the subject information input fromthe pattern detection unit 802, and the organ positions as well as theirregular region in the face model data input from the irregularitydetermination unit 804. In the present exemplary embodiment, there willbe described a method in which the normal generation unit 1301 performsprojective transformation of the face model data, in such a manner thatthe face region in the face model data becomes smaller than the faceregions in the captured image data.

FIG. 15A is a diagram illustrating an example of the captured imagedata, and FIG. 15B is a diagram illustrating an example of the facemodel data after the projective transformation. In the captured imagedata illustrated in FIG. 15A, the organ positions of a right eye 1501, aleft eye 1502, a nose 1503, and a mouth 1504, each indicated by a frameof a broken line, are moved from the original organ positionsillustrated in FIG. 11A toward the nose side. This can be implementedby, for example, moving each of the detected organ positions toward theorgan position of the nose at a certain ratio. In the face model dataafter the projective transformation illustrated in FIG. 15B, a faceregion 1506 after the projective transformation is smaller than a faceregion 1505 in the captured image data. This is because an effect ofreduction transformation centering on the organ position of the nose canbe obtained, by moving the organ position to be subjected to theprojective transformation, from the original organ position, toward thenose side. A reason for this transformation, i.e., the projectivetransformation performed so that the face region in the face model databecomes smaller than the face region in the captured image data, will bedescribed below in the description of processing to be performed by anormal correction unit 1303. The normal generation unit 1301 outputs thenormal image data generated by the above-described processing of theprojective transformation, to a mask generation unit 1302 and the normalcorrection unit 1303.

Next, in step S1402, the mask generation unit 1302 generates mask imagedata. FIG. 16A is a diagram illustrating an example of the normal imagedata, and FIG. 16B is a diagram illustrating an example of the maskimage data. FIG. 16A illustrates the normal image data generated in step1401, and FIG. 16B illustrates the mask image data calculated based onthe normal image data. The mask image data is binary image data in whicheach pixel is expressed as 0 or 1, and black is displayed as 0 whereaswhite is displayed as 1. In the present exemplary embodiment, a maskimage is generated in such a manner that each value in the backgroundregion of the normal image data is 0 and each value in the face regionis 1 in the mask image data. More specifically, the value of a pixelhaving a pixel value of (0, 1, 0) in the normal image data may beexpressed as 0 in the mask image data. The format of the mask image datais not limited to this type, and any format may be adopted if the formatincludes information capable of separating the background region and theface region of the normal image data. The mask generation unit 1302outputs the generated mask image data to the normal correction unit1303.

Lastly, in step S1403, the normal correction unit 1303 corrects thenormal image data based on the captured image data and the mask imagedata. In the present exemplary embodiment, the normal correction unit1303 corrects the normal image data, by using a cross bilateral filterbased on the pixel value of the captured image data, like the firstexemplary embodiment. The normal correction unit 1303 of the presentexemplary embodiment uses the mask image data input from the maskgeneration unit 1302, in this filtering processing. More specifically,the cross bilateral filter processing is performed using only pixelseach corresponding to the face region where the pixel values are 1 inthe mask image data. Therefore, the normal information of the backgroundis not used for the filtering in the outline region of the face region.Accordingly, a three-dimensional effect of the outline region can bemaintained, as compared with the processing in the first exemplaryembodiment.

Moreover, there is an effect of easy shaping the outline in the facemodel data according to the outline in the captured image data, byreducing the face region in the face model data to a size smaller thanthe face region in the captured image data. Therefore, the normal imagedata can be suitably corrected. The normal correction unit 1303 outputsthe processed normal image data to the lighting unit 305, and then theprocessing ends.

The outline of the processing performed in the normal acquisition unit303 according to the present exemplary embodiment has been describedabove. According to the-described processing, as compared with theprocessing in the first exemplary embodiment, the lighting processingthat maintains the three-dimensional effect of the outline region can beperformed.

In the second exemplary embodiment, there is described the method forcorrecting the normal image data based on the captured image data andthe mask image data. In a third exemplary embodiment, there will bedescribed an example in which the normal image data is corrected tomaintain a three-dimensional effect by further using the range imagedata, even if a shielding object is present in front of a subject.

FIG. 17 is a block diagram illustrating a configuration of the normalacquisition unit 303 according to the third exemplary embodiment.Further, FIG. 18 is a flowchart illustrating an operation of the normalacquisition unit 303 according to the third exemplary embodiment.Processing from step S901 to step S1401 is similar to the processingfrom step S901 to step S1401 in the second exemplary embodiment, andthus will not be described. Brief description will be provided, focusingon a point different from the second exemplary embodiment.

In step S1801, a distance acquisition unit 1701 acquires range imagedata from the distance acquisition unit 302. The range image data issimilar to the range image data of the first exemplary embodiment andthus will not be described. The distance acquisition unit 1701 outputsthe acquired range image data to a normal correction unit 1702.

Next, in step S1802, the normal correction unit 1702 corrects the normalimage data, based on the captured image data, the range image data, andthe subject information. FIG. 19A is a diagram illustrating an exampleof the captured image data. FIG. 19B is a diagram illustrating anexample of the range image data. FIGS. 19C and 19D are diagrams eachillustrating an example of the normal image data. FIG. 19A illustratesthe captured image data in which a shielding object 1902 is presentbetween a subject 1901 and the imaging apparatus 101, and a part of theface region of the subject is covered by the shielding object 1902. FIG.19B illustrates the range image data acquired in step S1801. In thisrange image data, distance information 1903 of the subject and distanceinformation 1904 of the shielding object are different. FIG. 19Cillustrates the normal image data, which is generated in step S1401 bythe projective transformation of the face model data. In the presentexemplary embodiment, the shielding object 1902 illustrated in FIG. 19Ashields the part of the face region of the subject 1901. However, thepresent exemplary embodiment is not limited to this example, and isapplicable to any case where a part of the face region of the subject iscovered. For example, even in a case where a part of the face region iscovered by a hand or body of the subject, processing similar toprocessing to be described below can be performed.

First, the normal correction unit 1702 determines whether a part of theface region of the subject is shielded by a shielding object. Thisdetermination can be performed using the subject information and therange image data. For example, the normal correction unit 1702 extractsthe face region of the subject by using the subject information, andrefers to the distance information of the region corresponding to theface region in the range image data. Next, the normal correction unit1702 calculates an average of the distance information in the faceregion. The normal correction unit 1702 then determines whether thenumber of pixels, each of which indicates a distance smaller than theaverage by an amount equal to or greater than a predetermined threshold,exceeds a predetermined number. It can be determined that the part ofthe face region of the subject is covered by the shielding object, ifthe number of pixels, each of which indicates the distance smaller thanthe average by the amount equal to or greater than the threshold,exceeds the predetermined number. In the present exemplary embodiment,the distance information 1904 of the shielding object, which isdifferent from the distance information 1903 of the subject, is includedin the face region. Therefore, it can be determined that the part of theface region of the subject is covered by the shielding object. In a casewhere no part of the face region of the subject is covered by theshielding object, processing similar to the processing performed by thenormal correction unit 1303 of the second exemplary embodiment may beperformed and therefore, further description is omitted.

Next, if the part of the face region of the subject is covered by theshielding object, the normal correction unit 1303 performs smoothingprocessing. The smoothing processing is performed in such a manner thatan intensity of smoothing in the shielded region corresponding to theshielding object is higher than an intensity of smoothing in otherregion in the face region. More specifically, for example, in theprocessing using a cross bilateral filter similar to that in the normalcorrection unit 806 of the first exemplary embodiment, the number oftaps of the filter is increased in the shielded region. If the maskprocessing in the second exemplary embodiment is performed, thesmoothing can be intensified by not performing the mask processing inthe shielded region. The method for increasing the degree of thesmoothing is not limited to the above-described methods, and othermethods may be used. For example, the shielded region may be extracted,and different types of smoothing processing may be applied to theshielded region and other regions in the face region, respectively.According to the processing described in the present exemplaryembodiment, there is no need to add new correction processing, andtherefore, the cost can be maintained to be low. FIG. 19D illustratesthe normal image data after the correction processing. In this normalimage data, the normal of a shielded region 1906 is highly smoothed.Such an area in which the normal is to be highly smoothed is not limitedto the shielded region as in the present exemplary embodiment. Forexample, in a case where the pattern detection unit 802 is capable ofobtaining reliability of a detection result of the organ positions ofparts such as eyes and a nose, the smoothing processing in the entireface region may be intensified when the reliability is low. The outlineof the processing performed in the normal correction unit 1303 of thepresent exemplary embodiment has been described above. According to theabove-described processing, an influence of the normal of the face modeldata on the shielding object can be suppressed, and more naturallighting processing can be performed.

The above-described exemplary embodiment is described using the examplein which the normal image data of a predetermined face is stored as theface model data in the ROM 203, and the position of the face model datais adjusted with that of the face detected in the captured image data.However, a case to which the exemplary embodiments are applicable is notlimited thereto. For example, the exemplary embodiments of thedisclosure are applicable to an example in which the normal image dataof an object, on which an identification code for position adjustment isprinted, is stored as three-dimensional model data. Position adjustmentof the three-dimensional model data is performed using an identificationcode detected in the captured image data. In this case as well, suitablelighting can be executed by performing position adjustment based on thesize of an irregularity around each of a plurality of reference pointsdetected in the identification code.

A fourth exemplary embodiment is directed to obtaining an image to whichshade and shadow suitable for an orientation and a facial expression ofa subject is added.

<External Appearance of Imaging Apparatus>

FIGS. 20A and 20B are diagrams illustrating an external appearance of animaging apparatus 1010 according to the present exemplary embodiment.FIG. 20A illustrates an appearance of a front face of the imagingapparatus 1010, and FIG. 20B illustrates an appearance of a back face ofthe imaging apparatus 1010. The imaging apparatus 1010 includes anoptical unit 1020, an imaging button 1030, an electronic flash 1040, adistance acquisition unit 1050, a display unit 1060, and an operationbutton 1070.

The optical unit 1020 is a lens barrel including a zoom lens, a focuslens, a shake correction lens, an iris, and a shutter, and collectslight information of a subject. The imaging button 1030 is provided fora user to instruct the imaging apparatus 1010 to start image capturing.The electronic flash 1040 is an illuminator capable of emitting lightwhen image capturing starts according to an instruction of the user. Thedistance acquisition unit 1050 is a distance acquisition module foracquiring range image data of a subject according to an image capturinginstruction. The range image data is image data in which, as a pixelvalue of each pixel of an image, a subject distance corresponding to thepixel is stored. The distance acquisition unit 1050 includes aninfrared-light emitting unit that emits an infrared light, and a lightreceiving unit that receives the infrared light reflected from asubject. The distance acquisition unit 1050 calculates a distance valueof a distance from the imaging apparatus 1010 to the subject, based onthe time from the emission of the infrared light to the receipt of theinfrared light reflected from the subject. The distance acquisition unit1050 then generates the range image data, by calculating positioninformation of the subject, based on the calculated distance value, anddistance imaging information including the number of sensor pixels andan angle of view of the light receiver. The method for acquiring therange image data is not limited to this method. For example, an opticalsystem similar to the optical unit 1020 may be provided in place of thedistance acquisition unit 1050, and the range image data may be acquiredby performing triangulation based on parallax between two pieces ofimage data captured from the respective two different viewpoints.

The display unit 1060 is a display such as a liquid crystal display,which displays image data processed in the imaging apparatus 1010 andother various kinds of data. In the present exemplary embodiment, anoptical viewfinder is not included in the imaging apparatus 1010 andtherefore, framing operation (confirmation of focus and composition) isperformed using the display unit 1060. In other words, the user performsimage capturing while confirming a live view image on the display unit1060 and therefore, the display unit 1060 may be said to serve as anelectronic viewfinder during the operation for framing and focusing.Besides displaying a live view of an imaging range in real time, thedisplay unit 1060 displays a camera setting menu.

The operation button 1070 is provided for the user to perform anoperation of changing an operation mode of the imaging apparatus 1010,and to specify various parameters required during image capturing in theimaging apparatus 1010. In the present exemplary embodiment, a lightingcorrection processing mode for correcting the degree of illumination ina captured image after image capturing, is included as one of theoperation modes. Using the operation button 1070 or the imaging button1030, the user can, for example, switch to the lighting correctionprocessing mode, set illumination parameters of virtual illumination tobe used for lighting correction, and select a subject for which thedegree of illumination is to be adjusted. Further, the user can providean instruction as to whether to output the range image data, whenoutputting the corrected image data. The display unit 1060 may have atouch screen function, and in such a case, a user instruction using atouch screen can be handled as an input of the operation button 1070.

<Internal Configuration of Imaging Apparatus>

FIG. 21 is a block diagram illustrating an internal configuration of theimaging apparatus 1010 according to the present exemplary embodiment.

A CPU 2020 is involved in processing of each configuration. The CPU 2020sequentially reads commands stored in a ROM 2030 and a RAM 2040, andinterprets the read commands, and executes processing according to theresult of the interpretation. A system bus 2120 is provided fortransmission and reception of data. In the present exemplary embodiment,it is assumed that the ROM 2030 stores a face normal model correspondingto a human face. The face normal model includes normal image data inwhich a normal vector of a face surface, which corresponds to a facehaving a predetermined shape, is stored in a pixel value. The facenormal model also includes organ position information indicating organpositions of parts such as eyes and a mouth of a person in the normalimage data.

A control unit 2060 is a control circuit that receives a userinstruction from the imaging button 1030 or the operation button 1070,and controls image capturing, switching to the lighting correctionprocessing mode, selecting a subject region, and setting illuminationparameters. An optical system control unit 2050 is a control circuitthat controls the optical unit 1020 to perform, for example, focusing,opening of the shutter, and adjustment of an aperture, instructed by theCPU 2020.

A color imaging sensor unit 2010 is an imaging sensor that convertslight information collected by the optical unit 1020 into a currentvalue. The color imaging sensor unit 2010 includes a color filter havinga predetermined array such as a Bayer array, and acquires subject colorinformation from the light collected by the optical unit 1020.

An A/D conversion unit 2080 is a processing circuit that converts thesubject color information detected by the color imaging sensor unit 2010into a digital signal value to have raw image data. In present exemplaryembodiment, the range image data and the raw image data that aresimultaneously captured can be acquired.

An image processing unit 2090 performs development processing on the rawimage data acquired by the A/D conversion unit 2080 to generate colorimage data. Using the color image data and the range image data, theimage processing unit 2090 performs various kinds of image processing,such as generating correction image data by correcting lighting in thecolor image data. An internal configuration of the image processing unit2090 will be described in detail below.

A character generation unit 2070 is a processing circuit that generatesa character and/or a graphic. The character and/or the graphic generatedby the character generation unit 2070 are superimposed on data such asthe image data and the correction image data, when displayed on thedisplay unit 1060.

An encoding unit 2100 performs processing for converting various kindsof image data, including the color image data processed by the imageprocessing unit 2090 and the correction image data generated by thelighting correction processing, into a file format such as JPEG.

A medium interface (I/F) 2110 is an interface for transmitting andreceiving image data to and from a PC/medium 2130 (e.g., a hard disk, amemory card, a compact flash memory card, and a secure digital card).For example, a universal serial bus (USB) is used for the medium I/F2110.

<Internal Configuration of Image Processing Unit>

FIG. 22 is a block diagram illustrating a functional configuration ofthe image processing unit 2090 according to the present exemplaryembodiment. A development unit 3010 generates color image data, byperforming processing, such as white balance processing, demosaicprocessing, noise reduction processing, color conversion processing,edge enhancement processing, and gamma processing, on the raw image dataacquired from the A/D conversion unit 2080. The generated color imagedata can be output to the display unit 1060 to be displayed thereby, andcan be stored into a storage device such as the RAM 2040 and thePC/medium 2130. In the present exemplary embodiment, the developmentunit 3010 generates the color image data without performing the gammaprocessing, and outputs the generated color image data to the lightingunit 3050.

A distance correction unit 3020 generates correction distance datacorresponding to a subject selected from the range image data, based oncolor image data, face information, and a subject position selected bythe user. In the present exemplary embodiment, it is assumed that thecorrection distance data stores a distance value mainly corresponding toa person corresponding to the selected subject position, and a distancevalue corresponding to a background that is a region except for theperson.

A face detection unit 3030 acquires face information of the subject fromthe color image data acquired from the development unit 3010. The faceinformation of the subject at least includes information about a faceregion indicating a region occupied by the face of the subject in thecolor image data, and organ positions indicating the positions of partssuch as eyes and a mouth included in the face, in the color image data.

A normal correction unit 3040 corrects the face normal model stored inthe ROM 2030, based on the face information acquired from the facedetection unit 3030 and the color image data acquired from thedevelopment unit 3010.

A lighting unit 3050 performs lighting processing on the color imagedata, based on the correction distance data acquired from the distancecorrection unit 3020, the correction normal data acquired from thenormal correction unit 3040, and the illumination parameters acquiredfrom the control unit 2060. The correction image data generated by thelighting processing can be output to a storage device such as the RAM2040 and the PC/medium 2130 to be stored therein, and can be output tothe display unit 1060 to be displayed thereby.

<Processing Flow of Image Processing Unit>

FIG. 23 is a flowchart illustrating an operation procedure of the imageprocessing unit 2090 in the imaging apparatus 1010 of the presentexemplary embodiment. In the present exemplary embodiment, the imageprocessing unit 2090 generates the correction distance datacorresponding to the selected subject, from the range image data, byusing the face information acquired from the color image data, and asubject position P0 acquired based on a user instruction. Next, based onthe face information of the subject, and the face normal model heldbeforehand, normal image data matching with the face of the subject isgenerated. The correction image data is subsequently generated byperforming the lighting processing for adding a virtual light source tothe color image data, based on the illumination parameters set by theuser operation, and the correction distance data as well as thegenerated normal image data. Details of the operation procedure of theimage processing unit 2090 will be described below.

In step S4010, the color image data is generated by performing thedevelopment processing such as the demosaic processing, on the raw imagedata acquired by the development unit 3010 from the A/D conversion unit2080. The color image data in the present exemplary embodiment will bedescribed with reference to FIG. 24A. In a pixel (i, j) of color imagedata I5010, it is assumed that RGB values are stored as a pixel value,and expressed as Ir(i, j), Ig(i, j), and Ib(i, j), respectively.However, the method for acquiring the color image data is not limited tothis method. For example, the development unit 3010 may generate colorimage data, by acquiring raw image data stored in the RAM 2040 or thePC/medium 2130. Alternatively, color image data already subjected to thedevelopment processing may be acquired from the RAM 2040 or thePC/medium 2130. Next, in step S4020, the development unit 3010 outputsthe color image data acquired in step S4010, to the display unit 1060.Based on the display on the display unit 1060, the user determineswhether to perform the lighting correction processing.

In step S4030, based on an input from the operation unit 1070, thecontrol unit 2060 determines whether an instruction to perform thelighting correction processing is input. When the instruction to performthe lighting correction processing is not input (NO in step S4030), theprocessing proceeds to step S4040. When an instruction to perform thelighting correction processing is input (YES in step S4030), the controlunit 2060 outputs a signal indicating that the lighting correction is tobe performed, to the development unit 3010 and the distance correctionunit 3020, and then the processing proceeds to step S4060.

In step S4040, the control unit 2060 determines whether an image outputinstruction is input by the user. When it is determined that an imageoutput instruction is input by the user (YES in step S4040), theprocessing proceeds to step S4050. When it is determined that an imageoutput instruction is not input by the user (NO in step S4040), theprocessing returns to step S4030.

In step S4050, the control unit 2060 outputs the image outputinstruction to the development unit 3010, and then, the development unit3010 outputs the color image data to the PC/medium 2130, and then theprocessing ends.

In step S4060, the distance correction unit 3020 acquires the rangeimage data from the distance acquisition unit 1050. The range image datain the present exemplary embodiment will be described with reference toFIG. 24B. It is assumed that a distance value D(i, j) representing adistance from the imaging apparatus 1010 to the subject is be stored asa pixel value, in a pixel (i, j) of range image data D5020. The methodfor acquiring the range image data is not limited to this method. Forexample, range image data stored in the RAM 2040 or the PC/medium 2130may be acquired.

In step S4070, the development unit 3010 outputs the color image data tothe face detection unit 3030, and the face detection unit 3030 acquiresthe face information of the subject from the input color image data. Theface information in the present exemplary embodiment will be describedwith reference to FIG. 25. The face information in the present exemplaryembodiment includes information indicating a face region 6010 and anorgan position 6020. The face region represents a set of pixels of aregion including the face, in color image data 5010. The organ position6020 represents coordinates corresponding to each of eyes and a mouth inthe face region. An existing algorithm may be used for the method fordetecting the face region and the organ position. Examples of thealgorithm include an algorithm using template matching and an algorithmusing a Haar-like feature amount. In the present exemplary embodiment,the face region and the organ position are detected by the templatematching. First, a flesh color region is extracted as a face candidateregion, by performing thresholding for the color image data. In otherwords, pixels, each of which has a pixel value falling in a pixel valuerange determined based on various flesh colors, are extracted as theface candidate region. Next, matching processing is performed for theface candidate region by using face image templates of various sizes,and calculates a likelihood of the face region. Finally, a region, inwhich the calculated likelihood is at a predetermined threshold orhigher, is extracted as the face region. The face detection unit 3030also performs similar template matching for the extracted face region,by using image templates of eyes and a mouth, thereby extractingcoordinates corresponding to each of the eyes and the mouth. The faceregion 6010 and the organ position 6020 are acquired by theabove-described processing. The face detection unit 3030 outputs theacquired face information, to the normal correction unit 3040. The organto be detected may be an organ such as a nose and ears, other than eyesand a mouth.

In step S4080, the distance correction unit 3020 determines the positionof the subject specified by the user. In the present exemplaryembodiment, the user specifies the position of a subject on which theuser desires to perform the lighting correction processing, by using atouch panel provided on the display unit 1060, or the operation button1070. The distance correction unit 3020 acquires, from the control unit2060, a subject selection position P0′ input by a user operation. Next,based on the acquired subject selection position P0′, the distancecorrection unit 3020 calculates a specified subject position P0 in thecolor image data. In the present exemplary embodiment, the color imagedata is displayed on the display unit 1060 having the touch screenfunction, and the display unit 1060 receives an operation of touchingthe subject on the display screen by the user. The distance correctionunit 3020 acquires the position touched by the user from the controlunit 2060, as the subject selection position P0′. In this process, thesubject selection position P0′ corresponds to a pixel position on thedisplay unit 1060. The distance correction unit 3020 calculates thesubject position P0, by converting this pixel position on the displayunit 1060, into a pixel position in the color image data.

In step S4090, by using the subject position P0 acquired in step S4080,and the color image data acquired from the development unit 3010, thedistance correction unit 3020 generates the correction distance datafrom the range image data acquired in step S4060. The details of thecorrection distance data generation processing will be described below.The distance correction unit 3020 outputs the generated correctiondistance data to the lighting unit 3050.

In step S4100, based on the face information acquired from the facedetection unit 3030, and the color image data input from the developmentunit 3010, the normal correction unit 3040 generates the correctionnormal data that is the normal image data matching with the face of thesubject. The details of the correction normal data generation processingwill be described below. The normal correction unit 3040 outputs thegenerated correction normal data to the lighting unit 3050.

In step S4110, based on the input correction distance data and the inputcorrection normal data, the lighting unit 3050 performs the lightingprocessing such as adding a virtual light source to the color imagedata, thereby generating the correction image data. Details of thelighting processing will be described below.

In step S4120, the lighting unit 3050 determines whether a change in thesetting of the illumination parameters to be used for the lightingprocessing is input from the control unit 2060. When it is determinedthat the setting of the illumination parameter is changed (YES in stepS4120), the processing returns to step S4110, and the lightingprocessing is performed again. When it is determined that setting of theillumination parameter is not changed (NO in step S4120), the processingproceeds to step S4130.

In step S4130, the lighting unit 3050 determines whether an image outputinstruction is input from the control unit 2060. When it is determinedthat an image output instruction is input (YES in step S4130), theprocessing proceeds to step S4140. When it is determined that an imageoutput instruction is not input (NO in step S4130), the processingreturns to step S4120. In step S4140, the lighting unit 3050 outputs thegenerated correction image data to the PC/medium 2130, and then theprocessing ends. The flow of the processing performed in the imageprocessing unit 2090 of the present exemplary embodiment has beendescribed above. According to the above-described processing, thelighting processing can be performed using the face normal modeltransformed according to the subject. Therefore, it is possible toobtain an image provided with natural shade suitable for the orientationand facial expression of the subject. The processing performed in eachpart of the image processing unit 2090 will be described in detailbelow.

<Correction Distance Data Generation Processing>

Here, the correction distance data generation processing performed bythe distance correction unit 3020 in step S4090 will be described withreference to a flowchart illustrated in FIG. 26. In step S7010, thedistance correction unit 3020 extracts the subject candidate region,based on the face information, the subject position P0, and the rangeimage data. The processing in this step will be described with referenceto FIGS. 27A and 27B. First, the distance correction unit 3020 selectsthe face region 6010 closest to the subject position P0, from the faceregion indicated by the face information. The distance correction unit3020 then acquires the distance value of each pixel in the selected faceregion 6010, from the range image data, and calculates an average of theacquired distance values, as a distance value of the face region. Then,the distance correction unit 3020 generates a binary image 8010, inwhich pixels are divided into pixels, each of which has a distance valuethat differs from the distance value of the face region 6010 by apredetermined threshold or less, and pixels other than such pixels. Inother words, the processing performed here is processing fordistinguishing a subject at a distance in a predetermined range ofdistances from the selected subject, from subjects other than such asubject. Here, in the binary image 8010, it is assumed that the pixels,each of which has the distance value that differs from the distancevalue of the face region 6010 by the predetermined threshold or less,form a subject candidate region 8020. The determination of the subjectcandidate region performed here is not limited to the above-describedmethod. A region, in which a distance value differs from the distancevalue of the selected subject position by a predetermined threshold orless, may be simply determined as the subject candidate region.

In step S7020, the distance correction unit 3020 performs shapingprocessing for removing a small connection component included in thesubject candidate region 8020 and for filling a hole, by performingsmall component removal processing and filling processing on the binaryimage 8010. Methods adoptable as the small component removal processingand the filling processing include a method using a morphologicaloperation and a method using labeling processing. In the presentexemplary embodiment, the method using the morphological operation isemployed. The distance correction unit 3020 performs opening processingon the subject candidate region 8020 included in the binary image 8010,as the small component removal processing. The distance correction unit3020 then performs closing processing on the subject candidate region8020, as the filling processing. FIG. 27C illustrates an example of abinary image 8030 obtained in this step.

In step S7030, the distance correction unit 3020 performs smoothingprocessing on the binary image 8030 subjected to the shaping processingin step S7020, thereby generating correction distance data 8040 (FIG.27D) that is multivalue data. For example, it is assumed that thesmoothing processing is performed on an image, in which the pixel valueof each of pixels included in the subject candidate region 8020 is 255,whereas the pixel value of each of other pixels is 0, in the binaryimage 8030. In this case, the correction distance data 8040 includingdistance information of 8 bits per pixel is generated. Further, it isassumed, in this process, the larger the pixel value is, the smaller thedistance to a subject is.

Filters adoptable for the smoothing processing include a Gaussianfilter, and a joint bilateral filter that performs smoothing whilereferring to pixel values of color image data. In the present exemplaryembodiment, it is assumed that a joint bilateral filter expressed by thefollowing expression (1) is used.

$\begin{matrix}{{O_{x} = {\frac{1}{k_{s}}{\sum\limits_{p \in \Omega}{{f\left( {p,s} \right)} \star {{g\left( {R_{p},R_{s}} \right)}I_{p}}}}}}{k_{s} = {\sum\limits_{p \in \Omega}{{f\left( {p - s} \right)} \star {g\left( {R_{p} - R_{s}} \right)}}}}{{f\left( {p,s} \right)} = {\exp \left( {- \frac{{p - s}}{2\sigma_{d}^{2}}} \right)}}{{g\left( {R_{p},R_{s}} \right)} = {\exp \left( {- \frac{Y\left( {R_{p},R_{s}} \right)}{2\sigma_{r}^{2}}} \right)}}} & (1)\end{matrix}$

In the expression (1), s represents a process target pixel, Ω representsa neighboring region of s, p represents a pixel included in Q, and Irepresents image data to be smoothed. Further, R represents referenceimage data, f represents a weight based on a distance between a pixel pand the pixel s, and g represents a weight based on a pixel value. Theweight f is set in such a manner that the larger the distance betweenthe pixel p and the pixel s is, the smaller the weight f is. The weightg is set in such a manner that the larger the difference between thepixel p of a reference image and the pixel s is, the smaller the weightg is. In the expression (1), it is assumed that Y expresses a luminancedifference in pixel value between the pixel p and the pixel s. In stepS7030, the smoothing processing is performed, using the binary image8030 as I, and the color image data as R. By using the joint bilateralfilter for the binary image 8030 while referring to the color imagedata, the smoothing processing can be performed using only the pixelshaving the pixel values close to each other in the color image data.Therefore, it is possible to perform the smoothing, while matching theoutline of the subject candidate region 8020, with the outline of thesubject in the color image data. The method for the smoothing processingis not limited to this method. For example, as a method for setting theweight f, equal weights within a neighboring region may be assigned.Further, as a method for setting the weight g, a weight based on a colordifference in place of the luminance difference may be assigned.Alternatively, for example, a constant weight, if a pixel value is aconstant value or less, may be assigned.

In addition, by acquiring the correction distance data 8040 as amultivalue image, strangeness of the subject outline region can bereduced in the lighting processing performed in step S4090. Byperforming the above-described processing, the distance correction unit3020 can acquire the correction distance data 8040, in which mainly afrontward subject and a background are separated and the distance valuescorresponding to the respective regions are stored. The filteringprocessing performed here is not necessarily the processing using thejoint bilateral filter, and any type of filtering processing may beemployed if the filtering processing is based on the pixel value of thecolor image data.

<Correction Normal Data Generation Processing>

Here, the correction normal data generation processing performed in thenormal correction unit 3040 in step S4100 will be described. Thecorrection normal data generation processing in the present exemplaryembodiment is processing for correcting the face normal model stored inthe ROM 2030 and the PC/medium 2130, based on the color image data.Details of the correction normal data generation processing will bedescribed below with reference to a flowchart illustrated in FIG. 28.

In step S9010, transformation parameters used in transforming the facenormal model according to the color image data are calculated. FIG. 29Aillustrates an example of the face normal information according to thepresent exemplary embodiment. The face normal model includes face normalimage data 10010, and organ position information 10020 corresponding tothe face normal image data 10010. The face normal image data 10010 isimage data in which a normal vector (Nx(i, j), Ny(i, j), Nz(i, j)) ofthe direction of the face is stored as a pixel value in a pixel N(i, j).Of the normal vector stored in the pixel (i, j), Nx(i, j), Ny(i, j), andNz(i, j) represent a component of an x-axis direction, a component of ay-axis direction, and a component of a z-axis direction, respectively.The x-axis, the y-axis, and the z-axis are orthogonal to each other.Further, it is assumed that all the normal vectors included in the facenormal image data 10010 are a unit vector. In a pixel corresponding tothe region of the face, it is assumed that a vector in a directionperpendicular to a face surface is stored as the normal vector. In apixel corresponding to the region except for the face, it is assumedthat a vector in a direction opposite to an optical axis of the imagingapparatus 1010 is stored as the normal vector. In the present exemplaryembodiment, it is assumed that the z-axis is opposite to the opticalaxis of the imaging apparatus 1010, and (0,0,1) is stored in the pixelcorresponding to the region except for the face, as the normal vector.The organ position information 10020 indicates a coordinate value ofeach of a right eye, a left eye, and a mouth in the face normal imagedata 10010.

In this step, based on the organ position information 10020corresponding to the face normal model, and the organ position 6020included in the face information of the color image data, the normalcorrection unit 3040 associates the coordinates of each of the righteye, the left eye, and the mouth in the color image data 5010, withthose in the face normal image data 10010. The normal correction unit3040 then calculates the transformation parameters for matching theorgan position information 10020 of the face normal image data 10010with the organ position 6020. As the transformation parameters, affinetransformation coefficients to be used for affine transformation arecalculated. A method such as a least squares method can be used as amethod for calculating the affine transformation coefficients. In otherwords, the affine transformation coefficients, which minimize a sum ofsquares of errors with respect to the organ position 6020 when theaffine transformation is performed on the organ position information10020, are determined as the transformation parameters at this time. Inthe present exemplary embodiment, the face normal image data 10010 hasthe components of the x-axis, y-axis, and z-axis directions of thenormal vector, as the pixel value. However, these may be assigned to,for example, the respective channels of 3-channel 8-bit color imagedata. For example, the component of each axis direction of the normalvector takes a value from −1.0 to 1.0, and therefore, information of thenormal vector can be held as the 3-channel 8-bit color image data, byassigning a value in this range to 0 to 255.

In step S9020, the normal correction unit 3040 generates the normalimage data 10030, by converting the face normal image data 10010 byusing the affine transformation coefficients calculated in step S9010.The normal image data 10030, in which the face normal image data 10010is fit into the face region included in the color image data 5010, isthereby generated. The normal image data 10030 is image data in which anormal vector (N′x(i, j), N′y(i, j), N′z(i, j)) is stored as a pixelvalue in a pixel N′(i, j). For a region corresponding to the face normalimage data 10010 (FIG. 29B), the normal vector of the normal image data10030 is calculated based on a normal vector (Nx, Ny, Nz) stored in eachpixel of the face normal image data 10010. For a region notcorresponding to the face normal image data 10010, it is assumed that anormal vector (0, 0, 1) in a direction opposite to the optical axis ofthe imaging apparatus 1010 is stored. By this step, the face region inthe face normal image data 10010 can be approximately matched with theface region in the color image data. However, positions other than theorgan positions, such as the outline of the face, may not be preciselymatched, and therefore, this is corrected in subsequent steps.

In step S9030, the normal correction unit 3040 divides the normal imagedata 10030 into the components of the x-axis, y-axis, and z-axisdirections, to have three kinds of image data, which are x-axiscomponent normal data 11010, y-axis component normal data 11020, andz-axis component normal data 11030 (FIG. 30A). Smoothing processingsimilar to that for the binary image 8030 can be thereby applied. In thepresent exemplary embodiment, the normal correction unit 3040 causes ajoint bilateral filter to act in a manner similar to step S7030.

In step S9040, the normal correction unit 3040 generates smoothed x-axiscomponent normal data 11040, by performing the smoothing processing onthe x-axis component normal data 11010. The joint bilateral filter,which uses the color image data 5010 as a reference image, is used forthe smoothing processing. In the smoothed x-axis component normal data11040 obtained by this processing, it is assumed that a value N″x of thesmoothed x-axis component is stored in each pixel.

In step S9050, the normal correction unit 3040 generates smoothed y-axiscomponent normal data 11050, by performing the smoothing processing onthe y-axis component normal data 11020. The joint bilateral filter,which uses the color image data 5010 as a reference image, is used forthe smoothing processing. In the smoothed y-axis component normal data11050 obtained by this processing, it is assumed that a value N″y of thesmoothed y-axis component is stored in each pixel.

In step S9060, the normal correction unit 3040 generates smoothed z-axiscomponent normal data 11060, by performing the smoothing processing onthe z-axis component normal data 11030. The joint bilateral filter,which uses the color image data 5010 as a reference image, is used forthe smoothing processing. In the smoothed z-axis component normal data11060 obtained by this processing, it is assumed that a value N″z of thesmoothed z-axis component is stored in each pixel.

By the processing in step S9040 to step S9060 described above, theoutline of the face in the normal image data 10030 can be matched withthe outline of the subject in the color image data.

In step S9070, the normal correction unit 3040 generates smoothed normalimage data 11070 (FIG. 30B), by integrating the smoothed x-axiscomponent normal data 11040, the smoothed y-axis component normal data11050, and the smoothed z-axis component normal data 11060. The smoothednormal image data 11070 is image data in which a normal vector (N″x(i,j), N″y(i, j), N″z(i, j)) is stored in a pixel (i, j).

In step S9080, the normal vector stored in each pixel of the smoothednormal image data 11070 is normalized to be the unit vector. In stepS9040 to step S9060, the smoothing processing is performed for each axiscomponent, and therefore, the normal vectors stored in the respectivepixels vary in magnitude. To correct this variation, the normal vectoris normalized to have a magnitude of 1 in this step, as expressed by anexpression (2).

$\begin{matrix}{{{N_{x}^{\prime\prime\prime}\left( {i,j} \right)} = \frac{N_{x}^{\prime\prime}\left( {i,j} \right)}{\sqrt{\left\{ {N_{x}^{\prime\prime}\left( {i,j} \right)} \right\}^{2} + \left\{ {N_{y}^{\prime\prime}\left( {i,j} \right)} \right\}^{2} + \left\{ {N_{z}^{\prime\prime}\left( {i,j} \right)} \right\}^{2}}}}{{N_{y}^{\prime\prime\prime}\left( {i,j} \right)} = \frac{N_{y}^{\prime\prime}\left( {i,j} \right)}{\sqrt{\left\{ {N_{x}^{\prime\prime}\left( {i,j} \right)} \right\}^{2} + \left\{ {N_{y}^{\prime\prime}\left( {i,j} \right)} \right\}^{2} + \left\{ {N_{z}^{\prime\prime}\left( {i,j} \right)} \right\}^{2}}}}{{N_{z}^{\prime\prime\prime}\left( {i,j} \right)} = \frac{N_{z}^{\prime\prime}\left( {i,j} \right)}{\sqrt{\left\{ {N_{x}^{\prime\prime}\left( {i,j} \right)} \right\}^{2} + \left\{ {N_{y}^{\prime\prime}\left( {i,j} \right)} \right\}^{2} + \left\{ {N_{z}^{\prime\prime}\left( {i,j} \right)} \right\}^{2}}}}} & (2)\end{matrix}$

Therefore, the correction normal data, in which a normal vector (N′″x(i,j), N′″y(i, j), N′″z(i, j)) having a magnitude of 1 is stored in a pixel(i, j), is acquired.

The normal correction unit 3040 thus acquires the correction normaldata. According to the above-described processing, the face normal modelcan be corrected to match with the face of the subject. Therefore,natural shade can be added to the face of the subject in the lightingprocessing. In addition, a great change in the normal direction due tothe smoothing processing can be prevented, by independently performingthe smoothing processing for each coordinate axis component as describedabove.

<Lighting Processing>

Here, the lighting processing performed in step S4110 will be described.The lighting processing in the present exemplary embodiment isprocessing for generating a correction image, by performing theprocessing of adding the virtual light source to the color image dataaccording to the illumination parameters set by the user operation,based on the correction distance data and the correction normal data.Details of the lighting processing will be described below withreference to a flowchart illustrated in FIG. 31.

In step S12010, the lighting unit 3050 acquires the illuminationparameters to be used for the lighting processing, which are set by theuser from the control unit 2060. In the present exemplary embodiment,the user sets a position Q, an orientation U, brightness α, and alight-source color L of virtual illumination, as the illuminationparameters, by operating the operation unit 1070.

In step S12020, the lighting unit 3050 corrects the pixel value of thecolor image data 5010, based on the correction distance data 8040, thenormal image data 1003, and the illumination parameters acquired in stepS11010. In the present exemplary embodiment, it is assumed thatcorrection image data I′ is generated by correcting the pixel value ofthe color image data according to an expression (3).

$\begin{matrix}{{{I_{r}^{\prime}\left( {i,j} \right)} = {{I_{r}\left( {i,j} \right)} + {\sum\limits_{m}{{k_{m}\left( {i,j} \right)}L_{r,m}{I_{r}\left( {i,j} \right)}}}}}{{I_{g}^{\prime}\left( {i,j} \right)} = {{I_{g}\left( {i,j} \right)} + {\sum\limits_{m}{{k_{m}\left( {i,j} \right)}L_{g,m}{I_{r}\left( {i,j} \right)}}}}}{{I_{b}^{\prime}\left( {i,j} \right)} = {{I_{b}\left( {i,j} \right)} + {\sum\limits_{m}{{k_{m}\left( {i,j} \right)}L_{b,m}{I_{r}\left( {i,j} \right)}}}}}} & (3)\end{matrix}$

Here, I′r, I′g, and I′b represent the pixel value of the correctionimage data I′, Lrm, Lgm, and Lbm represent the color of mthillumination, and km represents a correction degree for the pixel valuewith respect to the mth illumination. The correction degree km isdetermined based on the brightness α, the position Q, the orientation U,and a distance value as well as a normal vector V corresponding to thepixel (x, y), of the illumination. For example, km can be determined byan expression (4).

$\begin{matrix}{{k\left( {i,j} \right)} = {t\; \alpha \; {K(\rho)}\frac{{- {V\left( {i,j} \right)}} \cdot {N\left( {i,j} \right)}}{W\left( {{P\left( {i,j} \right)},Q} \right)}}} & (4)\end{matrix}$

The expression (4) will be described with reference to FIG. 32. In theexpression (4), t is a correction coefficient for adjusting thecorrection degree based on the virtual light source. In the presentexemplary embodiment, it is assumed that the correction coefficient tis 1. In the expression (4), α is a variable representing the brightnessof the illumination. Q is a vector representing the position of thelight source. P is a vector representing a three-dimensional position ofthe pixel (i, j), and calculated from the correction distance data 8040as follows. First, based on the pixel value of the correction distancedata 8040, a virtual distance value of a distance from the imagingapparatus 1010 to a subject position, corresponding to each pixel, iscalculated. It is assumed that, in this process, the larger the pixelvalue of a pixel in the correction distance data 8040 is, the smallerthe distance from the imaging apparatus 1010 to the pixel is. Next, thelighting unit 3050 calculates a three-dimensional position P of thepixel (i, j), based on the virtual distance value corresponding to eachpixel, and an angle of view of the imaging apparatus 1010 as well as animage size of the color image data 5010. W is a function that returns alarger value as the distance from the position P of the pixel (i, j) tothe position Q of the light source increases. Further, ρ represents anangle formed by a vector directed from Q toward P(i, j), and theorientation U of the illumination. K is a function that returns a largervalue as the angle ρ decreases. N(i, j) is a normal vector correspondingto the pixel (i, j), and V(i, j) is a unit vector representing adirection from Q toward P(i, j). By generating the correction image asdescribed in the present exemplary embodiment, the brightness can becorrected according to the position of the illumination and the shape ofthe subject. The lighting processing for performing addition for thepixel value according to the distance from the virtual light source isthus performed. By the above-described processing, a pixel, which iscloser to the virtual light source and has a smaller angle formed by thevector directed from the virtual light source toward the pixel (i, j)and the normal vector, can be corrected to become brighter. Therefore,as illustrated in FIG. 33, a correction image 14010, in which thesubject appears as if the subject is illuminated by the virtualillumination, can be obtained.

In step S12030, the lighting unit 3050 outputs the correction imagedata, in which the pixel value is corrected, to the display unit 1060 todisplay the correction image data, and then the processing ends. Then,the user views the correction image data displayed on the display unit1060, and inputs an illumination-parameter change instruction or animage output instruction.

According to the above-described processing, the lighting processing canbe performed using the face normal model transformed to match with thesubject. Therefore, it is possible to obtain an image provided withnatural shade suitable for the orientation and facial expression of thesubject.

In the present exemplary embodiment, the development unit 3010 serves asan image acquisition unit configured to acquire image data representingan image including a subject. The distance correction unit 3020 servesas a distance acquisition unit configured to acquire distanceinformation, which indicates a distance from an imaging apparatusimaging the subject to the subject, for each pixel of the image data.The distance correction unit 3020 also serves as a correction unitconfigured to correct the distance information. The face detection unit3030 serves as a detection unit configured to detect a face in the imagerepresented by the image data. The normal correction unit 3040 serves asa transformation unit configured to transform the face model data basedon the face detected by the detection unit, thereby generatingcorrection face data. The normal correction unit 3040 also serves as anormal acquisition unit configured to acquire normal informationindicating a normal direction on a surface of the subject for each pixelof the image data, based on the distance information. The lighting unit3050 serves as a processing unit configured to perform lightingprocessing for correcting a pixel value of the image data, based on thecorrection face data, the distance information, and the position of thelight source. The ROM 2030 serves as a holding unit configured to holdface model data representing a three-dimensional shape of apredetermined face.

The fourth exemplary embodiment is described using the example in whichthe pixel value of the color image data is corrected regardless of aluminance value of the subject. In a fifth exemplary embodiment, amethod for controlling a correction amount based on a luminance value ofa subject will be described. Controlling the correction amount based onthe luminance value of the subject can suppress overexposures (i.e.,phenomenon in which image detail and clarity is lost by imageprocessing), which occur if a region having a high luminance value iscorrected beforehand, and a noise increase that occurs if a dark part iscorrected.

The present exemplary embodiment is similar to the fourth exemplaryembodiment, in terms of the configuration of the imaging apparatus 1010and the basic flow of the processing, which therefore will not bedescribed again. The fifth exemplary embodiment is different from thefourth exemplary embodiment in that the correction coefficient tincluded in the expression (4) is determined based on the luminancevalue of each pixel, in lighting processing to be performed by thelighting unit 3050.

An example of the method for determining the correction coefficient t inthe present exemplary embodiment will be described with reference toFIGS. 34A and 34B. FIG. 34A illustrates an example in which thecorrection coefficient t is determined based on thresholds th1 and th2that are set beforehand. In this example, the correction coefficient tis determined to be t=1 in a section of 0≤Y<th1, a monotonous decreaseof t in a section of th1≤Y<th2, and t=0 in a section of th2≤Y, where Yis a luminance value of a pixel. When the correction coefficient t isthus determined, a pixel with a larger luminance value can have asmaller correction degree based on a virtual light source. Therefore, itis possible to obtain an effect of suppressing overexposures of a pixelwith a high luminance value due to an influence of the virtual lightsource. In FIG. 34A, the correction coefficient t is linearly decreasedto be a linear function of Y in the section of th1≤Y<th2. However, theway of decreasing the correction coefficient t is not limited to thisexample. When it is assumed that the correction coefficient t is thelinear function of Y, a correction image I′ is expressed as a quadraticfunction of a pixel value of color image data I. In this case, a pixelvalue of the correction image may be maximized in the section ofth1≤Y<th2, and a tone reversal may therefore occur. To suppress thissituation, the decrease in the section of th1≤Y<th2 may be expressed,for example, by a quadratic curve or a trigonometric function. This cansuppress the occurrence of the tone reversal in the correction image.Alternatively, the correction coefficient t can be determined asillustrated in FIG. 34B. FIG. 34B illustrates an example in which thecorrection coefficient t is determined based on thresholds th1, th2, andth3 set beforehand. In this example, the correction coefficient t isdetermined to achieve a monotonous increase of t in a section of0≤Y<th3, t=1 in a section of th3≤Y<th1, a monotonous decrease of t in asection of th1≤Y<th2, and t=0 in a section of th2≤Y, where Y is aluminance value. Setting the correction coefficient t as illustrated inFIG. 34B can suppress, in addition to overexposures, emphasis of noiseof a dark part with a small luminance value due to the lightingprocessing. The increase in the section of 0≤Y<th3, or the decrease inthe section of th1≤Y<th2 may be expressed by a quadratic curve or atrigonometric function. In this case, occurrence of a tone reversal canbe suppressed as in the case illustrated in FIG. 34A.

As described above, according to the processing in the present exemplaryembodiment, it is possible to suppress overexposures, which occur if aregion having a high luminance value is corrected beforehand, and anoise increase that occurs if a dark part is corrected.

The fourth exemplary embodiment and the fifth exemplary embodiment areeach described using the example in which the lighting processing isperformed to brighten a subject appearing dark, by adding the virtuallight source to the scene. In a sixth exemplary embodiment, there willbe described a method for emphasizing a three-dimensional effect of asubject, by adding shade to a subject that appears flat due to aninfluence of an electronic flash.

The present exemplary embodiment is similar to the fourth exemplaryembodiment, in terms of the configuration of the imaging apparatus 1010and the basic flow of the processing, which therefore will not bedescribed again. The sixth exemplary embodiment is different from thefourth exemplary embodiment, in terms of the processing for correctingthe pixel value performed in step S12020. The processing to be performedin step S12020 of the present exemplary embodiment will be describedbelow. In step S12020 of the present exemplary embodiment, the pixelvalue is corrected based on the following expression (5), unlike thefourth exemplary embodiment.

$\begin{matrix}{{{{I_{r}^{\prime}\left( {i,j} \right)} = {{I_{r}\left( {i,j} \right)} - {\sum\limits_{m}{{k_{m}^{\prime}\left( {i,j} \right)}L_{r,m}{I_{r}\left( {i,j} \right)}}}}}{I_{g}^{\prime}\left( {i,j} \right)} = {{I_{g}\left( {i,j} \right)} - {\sum\limits_{m}{{k_{m}^{\prime}\left( {i,j} \right)}L_{g,m}{I_{r}\left( {i,j} \right)}}}}}{{I_{b}^{\prime}\left( {i,j} \right)} = {{I_{b}\left( {i,j} \right)} - {\sum\limits_{m}{{k_{m}^{\prime}\left( {i,j} \right)}L_{b,m}{I_{r}\left( {i,j} \right)}}}}}} & (5)\end{matrix}$

The expression (5) is difference from the expression (3) in that thepixel value is corrected so as to decrease the pixel value of the colorimage data according to a correction degree k′m. In other words, theprocessing performed in the present exemplary embodiment is lightingprocessing for performing subtraction for the pixel value according to adistance from a virtual light source. The correction degree k′m isdetermined based on brightness α, a position Q, and an orientation U ofillumination, as well as a distance value and a normal vector Vcorresponding to a pixel (x, y). For example, the correction degree k′mis determined by an expression (6).

$\begin{matrix}{{k^{\prime}\left( {i,j} \right)} = {t\; \alpha \; {K(\rho)}\frac{1 + {{V\left( {i,j} \right)} \cdot {N\left( {i,j} \right)}}}{W\left( {{P\left( {i,j} \right)},Q} \right)}}} & (6)\end{matrix}$

The expression (6) is different from the expression (4), mainly in thatthere is an influence of an angle formed by normal vectors N(i, j) andV(i, j). In the expression (4), the more directly the normal vector Nfaces the virtual light source, the larger the value of k is. Incontrast, in the expression (6), the more directly the normal vector Nfaces the virtual light source, the smaller the value of k is. In otherwords, the expression (6) can add stronger shade to a pixel, which iscloser to virtual illumination and has the normal vector N less directlyfacing the virtual light source. Therefore, as represented by acorrection image 16010 illustrated in FIG. 35, shade can be added onlyto a cheek and a nose of a face based on the normal image data.

According to the above-described processing, it is possible to performthe correction for adding shade for producing a three-dimensionaleffect, to a subject that appears flat due to an influence of anelectronic-flash emission.

In the above-described exemplary embodiments, there are described thelighting processing for adding the virtual light source to the scene,and the lighting processing for adding the shade to the image. In aseventh exemplary embodiment, there will be described a method forswitching between the above-described two kinds of processing based onimage capturing conditions. FIG. 36 is a flowchart illustrating anoperation procedure of the image processing unit 2090 in the seventhexemplary embodiment. The seventh exemplary embodiment is different fromthe fourth exemplary embodiment in that steps S17010 and S17020 areadded.

In step S17010, the lighting unit 3050 acquires information about anactual light source. In the present exemplary embodiment, the actuallight source is a light source that is actually present in a space wherean image of a subject is captured. In the present exemplary embodiment,it is assumed that the control unit 2060 determines the presence orabsence of an electronic-flash emission, based on an instruction forusing the electronic flash 1040 from the user, or an input signal fromthe electronic flash 1040. The control unit 2060 then outputs, to thelighting unit 3050, information indicating whether the electronic flash1040 is used in image capturing for obtaining image data. When it isdetermined that the electronic flash 1040 is used in image capturing forobtaining image data, information, which indicates light being emittedby the electronic flash 1040 located at a predetermined position Q′, isacquired. The method for acquiring the information about the actuallight source is not limited to this method. For example, an averageluminance of pixels in a face region of a selected subject may becalculated, and occurrence of an electronic-flash emission may bedetermined when the average luminance is at a threshold or higher.

In step S17020, the lighting unit 3050 sets a mode for the lightingprocessing, based the actual light source information acquired in stepS17010. When the absence of an electronic-flash emission in imagecapturing is determined in step S17010, a lighting mode corresponding tothe lighting for adding the virtual light source described in the fourthexemplary embodiment is set in step S17020. When the presence of anelectronic-flash emission in image capturing is determined in stepS17010, a lighting mode corresponding to the lighting for adding theshade described in the sixth exemplary embodiment is set in step S17020.

Next, in step S4110, the lighting unit 3050 generates the correctionimage data, by performing the lighting processing corresponding to thelighting mode set in step S17020, on the color image data.

The flow of the processing according to the present exemplary embodimenthas been described above. According to the above-described processing,appropriate lighting processing can be selected according to the stateof a light source when an image of a subject is captured.

The processing in the present exemplary embodiment is not limited to theabove-described example. For example, the position Q′ of the electronicflash light may be acquired as the actual light source information instep S17010, and the position Q′ of the electronic flash light may beinput as the initial value of the illumination parameter in the lightingprocessing in step S4110. Alternatively, the lighting mode may be set asfollows. First, an actual light source other than the electronic flashis detected in a region where a luminance is larger than a predeterminedthreshold in the color image data. When the detected actual light sourceis present at a position closer to the imaging apparatus 1010 than to asubject, the lighting mode for adding the shade is set. The position ofan actual light source may be acquired from the color image data, andinput as the initial value of the illumination parameter.

Other Exemplary Embodiments

An exemplary embodiment of the disclosure is not limited to theabove-described embodiments. For example, the lighting processing may beperformed by directly using the distance information of a subject,without using the normal image data. In this case, it is necessary touse a calculation expression different from the above-describedexpressions and thus, the processing becomes complicated. However, aneffect similar to those in the above-described exemplary embodiments ofthe disclosure can be obtained. In this case, a 3D model of apredetermined face may be held in place of the face normal model. Inother words, information indicating a three-dimensional shape of asubject may be widely used in an exemplary embodiment of the disclosure.Further, a 3D model of a predetermined face may be held in place of theface normal model, and the normal information may be acquired based onthe 3D model after transforming the 3D model.

The above-described exemplary embodiments can reduce strangeness of animage or render the image normal when image lighting correction isperformed using three-dimensional shape data.

Other Embodiments

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

What is claimed is:
 1. An image processing apparatus comprising: anacquisition unit configured to acquire a captured image obtained bycapturing a object; a specifying unit configured to specify a regionincluding in at least part of the object in the captured image; and ageneration unit configured to generate a correction image in whichbrightness of at least part of the specified region in the capturedimage is changed using pixel values of pixels in the captured image. 2.The image processing apparatus according to claim 1, wherein thegeneration unit generates, as the correction image, an image in whichthe brightness is changed in accordance with irregularity of at leastpart of the object.
 3. The image processing apparatus according to claim1, wherein the correction image is generated such that a degree ofchange of brightness in the specified region is larger than a degree ofchange of brightness in a background region excluding the object whoseregion has been specified by the specifying unit and excluding a regionnear the subject.
 4. The image processing apparatus according to claim1, wherein the specifying unit specifies a face of a person as theregion in the object in the captured image.
 5. The image processingapparatus according to claim 4, wherein the generation unit generates,as the correction image, an image in which brightness of a region of aobject that is not a person in the captured image is not changed.
 6. Theimage processing apparatus according to claim 1, wherein a degree ofchange from brightness of the specified region in the captured image tobrightness of the specified region in the correction image correspondsto the brightness of the specified region in the captured image.
 7. Theimage processing apparatus according to claim 1, wherein a degree ofchange of brightness of a first pixel from the captured image to thecorrection image is different from a degree of change of brightness of asecond pixel from the captured image to the correction image, whereinthe first pixel and the second pixel are included in the specifiedregion, and wherein a distance from an imaging device at the first pixelis equal to a distance from the imaging device at the second pixel, anda normal direction corresponding to the first pixel is different from anormal direction corresponding to the second pixel.
 8. The imageprocessing apparatus according to claim 1, wherein the generation unitdoes not change brightness in accordance with a normal direction of theobject in a region excluding the specified region specified by thespecifying unit and also excluding a region near the specified region.9. The image processing apparatus according to claim 1, wherein thespecifying unit specifies a region of a face of the subject.
 10. Animage processing method for generating a correction image in whichbrightness of a captured image acquired by capturing a object ischanged, the image processing method comprising: acquiring the capturedimage; specifying a region including in at least part of the object inthe captured image; and generating the correction image in whichbrightness of at least part of the specified region in the capturedimage is changed using pixel values of pixels in the captured image. 11.A non-transitory computer-readable storage medium storing instructionsthat, when executed by a computer, cause the computer to perform amethod for generating a correction image in which brightness of acaptured image acquired by capturing a object is changed, the imageprocessing method comprising: acquiring the captured image; specifying aregion including in at least part of the object in the captured image;and generating the correction image in which brightness of at least partof the specified region in the captured image is changed using pixelvalues of pixels in the captured image.